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ABSTRACT 

A method is presented which provides for increased 
subject involvement in the research design along with greater 
protection of the subjects* rights of privacy. An experimental 
analysis revealed that with increased motivation subjects took longer 
to complete the questionnaire; obtained higher, and less socially 
desirable. Dogmatism scores; and were less likely to manifest a 
response style which was found to be a function of answer sheet 
configuration. However, even with increased motivation, experimenter 
effects were found to produce a significant bias. Implications for 
attitude assessment research in the military are discussed. (Author) 
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Gilbert R. Kaats 
United States Air Force Academy 



Abstract . A method is presented which provides for 
increased subject involvement in the research design along 
with greater protection of the subjects’ rights of privacy. 
An experimental analysis revealed that with increased 
motivation subjects took longer to complete the question- 
naire; obtained higher, and less socially desirable, Dog- 
mat ism scores; and were less likely to manifest a response 
style which was found to be a function of answer sheet 
configuration. However, even with increased motivation, 
experimenter effects were found to produce a significant 
bias. Implications for attitude assessment research in 
the military are discussed. 



The continuing concern for respecting subjects" rights of privacy is 
particularly important when using military personnel as subjects. The 
basic authority structure of the military may provide an added degree of 
influence or pressure for subjects to answer items they might otherwise 
leave blank. Methods need to be employed to offset this influence. This 
research presents a method designed to accomplish this along with increasing 
subject involvement in the assessment process. An experimental analysis 
of the effects of this increased motivation on scale scores, response style, 
experimenter bias, scale reliability, and the. use of different answer sheet 
configurations is presented. 



One of the major problems facing researchers using written questionnaires 
is the unmotivated respondent who either fails to respond or, when he does, 
gives answers which are manifestations of carelessness and disinterest rather 
than basic attitudes or beliefs. While the use of anonymous questionnaires 
has traditionally been employed to increase candor, it provides a condition 
of relatively low subject involvement, precludes comparisons with other data 
on the subjects, and impedes follow-up research. The signed questionnaire 
increases personal involvement but may suffer from distortions due to social 
desirability or distrust of the expressed aims of the research. While the 
literature on the effects of the two approaches is mixed (see Hamel & Reif, 
1952; Rosen, 1960), it does appear the two procedures can lead to different 
results. 



Alternative procedures have sometimes been employed involving the 
surreptitious identification of subjects for data gathered under supposedly 
anonymous conditions. Not only do these techniques raise ethical questions, 
but they also may increase distrust among an already suspicious subject 
population. Recent attempts to resolve this dilemma have been to appeal to 



Vrhe views expressed herein are those of the author and do not necessarily 
reflect.. the official views of the Air Force Academy or the Department of 
Defense . 
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the subject beforehand, attempting to increase his involvement by explaining 
the general aims of the research, insuring confidentiality, and explaining 
that since inventories may contain •’personal*' and ‘'controversial" material, 
one should think carefully before completing them (Lovell, 1967). However, 
just exactly what does "personal" and "controversial" mean? Some respondents 
may agree or refuse to participate because of misinterpretations of what this 
actually means or a belief that they will not receive socially undesirable 
scores. Consequently it is doubtful that such a procedure provides the respon- 
dent with sufficient information to make an informed decision and precludes 
analysis of volunteer effects. An alternative procedure would be to give the 
respondent feedback on what was measured and how he scored on the measures 
before he was asked to sign the questionnaire. The method presented below 
suggests such a procedure with the added benefits of increased subject motiva- 
tion and a provision for analysis of volunteer effects. But increased motiva- 
tion raises questions as to its effects on response styles, experimenter bias, 
and assessment of socially undesirable personality factors. 

With respect to response style, this study focussed on a response style 
associated with different answer sheet configurations. Pilot research using 
answer sheets with the high - score category (strongly agree) to the right 
appeared to produce higher scores when compared to a condition where responses 
were written alongside each item. This suggested a type of response style where 
subjects favored the right-hand side of the answer sheet when responding under 
conditions of anonymity. To investigate the possibility that this effect was 
a function of a low level of motivation, subject motivation was varied along 
with four variations in answer sheet configuration. 



The following hypotheses were investigated: (1) A right-hand response 

style will elevate scores on answer sheets with the high score category to the 
right, attenuate scores with the high score category to the left, and have no 
effect on a counterbalanced configuration. (2) As subject motivation increases, 
the right-hand response style will decrease and time to complete the question- 
naire will increase. (3) The combined conditions of anonymity and feedback 
will encourage candor and lead to higher, and less socially desirable, scores. 
Finally, a condition was added involving the use of a personal briefing by 
instructors to provide for an analysis of experimenter bias at high levels of 
subject motivation. 

Method 

Subjects . All Ss (N-480) were freshmen and sophomores at the Air Force 
Academy enrolled in a required introductory psychology course. 

Procedure. In a completely counterbalanced design, 30 Ss were assigned 
to each of 16 cells in a 4 X 4 design. All cells were equated on a measure 
of intellectual ability (Verbal Aptitude Scores on the College Board Examina- 
tions); Ss used the same textbook, took basically the same examinations, and 
were assigned grades based on their standing relative to all cadets enrolled 
in the course. Instructors were randomly assigned to the 16 cells, and Ss 
completed Rokeach“s 40-item Dogmatism scale under four conditions of motiva- 
tion. In condition 1, the scale was completed anonymously without a promise 
of feedback. All instructions were written on the questionnaire itself and 
instructor involvement was minimal. In condition 2, the same procedure was 
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followed except the written instructions asked subjects to place their mili- 
tary serial numbers on the answer sheet* In condition 3, the written instruc- 
tions explained an anonymous- feedback method where each S picked a four-digit 
number at random and recorded it on his answer sheet. Answer sheets were 
scored and returned using the four-digit number known only to the J5. At this 
time a briefing was conducted explaining what the score represented along with 
normative data on cadet and civilian college populations. All Ss were cautioned 
that results should be viewed tentatively due to imperfect scale validity and 
the possibility of their having made mistakes in the recording of answers. It 
was at this time that the purposes of the research were explained and £s were 
asked to voluntarily record their serial numbers on the answer sheets. Thus, 
before signing the questionnaire, Ss knew what had been measured, how they 
scored, and how the results were to be used. Finally, condition 4 was designed 
to maximize instructor effects while retaining all other elements of condition 
3. In addition to providing written instructions, each instructor explained 
the anonymous- feedback system and solicited cooperation indicating that he was 
personally involved in the research. 

Four different answer sheet configurations were used; "strongly agree" 
to the left, "strongly agree" to the right, a "counterbalanced" pattern where 
the "strongly agree" category was alternated between left and right, and a 
"write-in" configuration where responses were recorded directly on the ques- 
tionnaire itself. 
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Results and Discussion 




As indicated in Table 1 and in support of hypothesis 1, the difference j 

between cells 1A and IB was highly significant (p<*01) suggesting the 1 
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Table 1 

Means on Rokeach°s 40-Item Dogmatism Scale--Four Levels | 

of Motivation & Four Types of Answer Sheets (N«480) 




Motivation condition 



Type 

of 

Answer 

Sheet 


(1) 

Anon 


(2) 

Signed 


(3) 

Anon 

•f Feedback 


(4) 

Anon 

+ Feedback 
+Briefine 


(A) Agree left 


127.0 


130.4 


144.0 


128.9 


(B) Agree right 


137.6 


129.4 


136.0 


128.9 


(C) Counterbalanced 


130.9 


131.3 


138.4 


133.9 


(D) Write-in 


133.5 


135.1 


141.4 


125.7 


Column means 


132.3 


131.5 


140.0 


129.4 


Minutes to complete 


10.96 


10.99 


11.64 


12.38 




t 



k 



I 







4 






agree« left configuration yielded lower scores, the agree-right higher scores, 
and both counterbalanced and write-in configurations yielded scores in between 
the two. Consistent with hypothesis 2, as motivation increased, this type of 
response style disappeared- -in fact, the difference between the agree-left and 
agree-right configurations was significant only in the anonymous no-feedback 
condition. Also, as hypothesized, an analysis of variance between the four 
motivation conditions and time to complete the questionnaire produced a sig- 
nificant F-ratio (p<.05). Completion times progressively increased across 
motivation conditions, and the difference between conditions 1 and 4 was highly 
significant (p<«01). With respect to hypothesis 3, an analysis of variance 
between motivation level and Dogmatism score produced a highly significant 
F-ratio (p<>001). Comparison of the column means reveals that while the use 
of anonymous or signed questionnaires made little difference, the anonymous 
feedback condition (3) yielded significantly higher (p<*01) and less socially 
desirable Dogmatism scores. 

When experimenter effects were maximized in condition 4, scores were 
significantly lower (p<.01) than in condition 3. This suggested that even 
under conditions of high involvement, subjects may have completed the ques- 
tionnaire in a manner in which they felt their instructors would approve- 
To test this notion, Ss anonymously rated their instructors on a locally 
developed dogmatism scale (see Kaats & Thompson, 1968). Examination of these 
ratings revealed that all but one instructor was perceived as being nondog- 
matic, suggesting that, as a group, instructors in condition 4 way have provided 
an influence toward lowered scores. Furthermore, when the subject means for 
sections where instructors were rated ''high” on dogmatism were compared with 
those sections where instructors were rated low, the average score on the 
Rokeach scale was significantly (p<.01) higher in the *'high M group than in 
the "loW" group. Thus, support was provided for the view that subject Dogmatism 
scores were influenced by the level of dogmatism seen in the instructor.. 

. - i 

Calculation of the average scale reliabilities for each motivation condi- 
tion revealed that both of the feedback conditions (3 & 4) yielded lower 
reliability coefficients than either of the no-feedback conditions (1 & 2). 

Since the Rokeach scale has no reversed items, this may suggest that with 
increased involvement, subjects made more differentiations between items in 
this domain. Finally, an analysis of volunteer effects was possible since 
the questionnaire was completed before Ss were asked to volunteer their names 
Only 2 \ refused to sign the answer sheet and, with respect to mean scores, 
they were no different from those who did sign. 

These results would appear to question the wisdom o£ employing anonymous/ 
no-feedback conditions in attitude assessment research and offer an alternative 
method with added ethical benefits and increased subject involvement. 



^In research not reported here both dogmatism and items on the Rokeach scale 
were rated by cadets as being socially undesirable. 
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